Triangulations of nearly convex polygons *^ 

Roland Bacher, Frederic Mouton 
December 13, 2010 



Abstract. — Counting Euclidean triangulations with vertices in a finite set C of the convex hull 
Conv(c) ofC is difficult in general, both algorithmically and theoretically. The aim of this paper 
is to describe nearly convex polygons, a class of configurations for which this problem can be 
solved to some extent. Loosely speaking, a nearly convex polygon is an infinitesimal perturbation 
of a weakly convex polygon (a convex polygon with edges subdivided by additional points). Our 
main result shows that the triangulation polynomial, enumerating all triangulations of a nearly 
convex polygon, is defined in a straightforward way in terms of polynomials associated to the 
"perturbed" edges. 



1 Introduction 

Given a finite subset C of the Euclidean plane R^, calculating the number of triangu- 
lations of the convex hull Conv(c) using only Euclidean triangles with vertices in C 
seems to be difficult and has attracted some interest, both from an algorithmic and a 
theoretical point of view, see for instance ifTI. 121. Il3l. Il4l. Il5l. HTl. 191. ifTOl. ifTTI. 

An important and well understood special case is given by the n vertices of a strictly 
convex polygon. The associated number of triangulations is the Catalan number C„_2- 

In a first part of the paper, we consider convex polygons having perhaps collinear 
vertices, called weakly convex polygons. We are not aware of the existence of formulae 
giving the number of triangulations for such polygons. Thinking of weakly convex 
polygons as strictly convex polygons with edges subdivided by additional vertices, we 
call edges the edges of the underlying strictly convex polygon. Edges are thus maximal 
straight segments contained in the boundary of such polygons. 

Denoting by C the set of all vertices, we define the weight of an edge E as the 
number of connected components of £ \ (£ n C ). Weights of successive edges form a 
finite sequence a\, 02... ai of total sum n=#C . 

We show (Theorem 13. 2I 1 that there exists a sequence of polynomials Pm{t), m>\, 
called maximal edge polynomials, such that the number of maximal triangulations {i.e. 
involving all vertices of C) equals 



X 



max 



{C)^Y.^kCk-2. (1) 



k>2 



where the coefficients are defined by n!=i Pn/CO ^ Y.k^kt'^- 

We deduce that the triangulation polynomial of a configuration (which takes into 
account non-maximal triangulations) verifies formally the same formula as the previous 
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one, replacing maximal edge polynomials by complete edge polynomials. This has the 
perhaps surprising consequence that enumerative properties of triangulations do not 
depend on the particular cyclic order of the edges. 

In a second part, we define nearly convex polygons as small perturbations of weakly 
convex ones. Our main result. Theorem 14.51 establishes the existence of near-edge 
polynomials such that the previous formulae continue to hold. Factorization of near- 
edges, a useful arithmetical property, allows classification of small near-edges. 

Near-edge polynomials are difficult to compute in general except in a few special 
cases. We plan to describe algorithms in a future paper dealing with computation as- 
pects. Some details are given in section|5] 

This article is organized as follows. We introduce first definitions and recall the 
strictly convex case in section |2] prove formulae for weakly convex polygons in sec- 
tion[3] expose the nearly convex setting and prove our main result in sectionH] Finally, 
section |5]contains a few remarks and open problems. 

2 Triangulations of planar configurations 

A (planar) configuration of points is a finite subset C = {Pi,...,P„} of the oriented 
plane R^. We denote by Conv(c) the convex hull of C and by Extr(c) the set of all 
extremal elements in C (a point P E C is, extremal if Com{c \ {P}) 7^ Conv(c)). The 
configuration C is said to be strictly convex if Extr(c) = C- More generally, Extr(c) 
is the set of vertices of the strictly convex polygon formed by the convex hull of C ■ 

A triangulation of a configuration C is a triangulation of its convex hull with ver- 
tices in C, i.e. a finite set T = {Ai, . . . ,A^} of Euclidean triangles with vertices in C 
such that Conv(c) — U^'^j A, and non-trivial intersections A, n A^ consist of a common 
vertex or a common edge. A triangulation of C is maximal if it involves all vertices of 
C (i.e. each point of C is a vertex of at least one triangle). The number of maximal 
triangulations of C is denoted by Tmax(c). If C is strictly convex, it is well known that 
(C) is a Catalan number. 

Theorem 2.1 All triangulations of a strictly convex n—gon are maximal and their 
number is C„_2 = C^"l2 ^) /(" ^ !)• 

Sketch of proof: A deformation argument shows that combinatorial properties of tri- 
angulations of a strictly convex «— gon depend only on n. We denote by T„ the number 
of triangulations of such a convex «— gon P. The choice of a marked edge £ in P se- 
lects in every triangulation a unique triangle A containing E. The two remaining edges 
of A determine two triangulated convex polygons having respectively k and n + l — k 
edges for some integer k such that 2 < k < n — I. This decomposition amounts to the 
recurrence relation 

n-l 
k=2 

holding for n > 3, using the convention T2 = 1- Therefore, the generating function 
Ln=2'^n-^" satisfies a quadratic equation. The classical resolution gives a formula, 
whose development in power series yields the result. □ 
Triangulations of a general configuration C are not necessarily maximal and enu- 
merative properties are encoded by the triangulation polynomial Px(c) = '^lk{c)s'^, 
where Ti(c) counts the number of triangulations using exactly k points. The polyno- 
mial Pz{c) has degree « —#C, with leading coefficient T„(c) = Tmax(c) counting the 
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number of maximal triangulations. Its monomial of lowest degree m = #Extr(c ) corre- 
sponds to the C„,_2 triangulations of the convex hull Conv(c) involving only extremal 
vertices. Remark also that the average number of points of a triangulation is given by 
the logarithmic derivative of the triangulation polynomial f{s) = f x(c). 

Two configurations are isotopic if they are related by a continuous deformation 
which preserves collinearity and non-collinearity of triplets. Isotopic configurations 
have the same triangulation polynomial. 

3 Weakly convex polygons 
3.1 Definition and notations 

A configuration C is weakly convex if it is contained in the boundary 8Conv(c) of its 
convex hull. We also call C a weakly convex polygon. We are not aware of a published 
formula giving the number of triangulations of such polygons. 




Figure 1: Two weakly convex polygons with edge-weights 1,5,2,3,4. 

Weakly convex polygons can be seen as strictly convex polygons with additional 
vertices subdividing their edges. We thus call edges the segments joining two consec- 
utive extremal vertices of the underlying strictly convex polygon. An edge has weight 
a if it involves a + 1 points of C ■ The weights of consecutive edges, in counterclock- 
wise order, define, up to cyclic permutations, a finite sequence a\, a2--- ai of total sum 
n = #C (see Figure[T]i- This sequence characterizes C up to isotopy. Thus all combina- 
torial properties of triangulations depend only on the sequence a\, a2--- ai (up to cyclic 
permutations). We denote by Tjt(ai ,a2, ■ • • the number of corresponding triangu- 
lations using exactly k points of C- This number is non-zero only for I < k < n. We 
denote the triangulation polynomial of C by px i , fl2 ,•••,«/) ■ 

The following notations will be useful. The number T„(ai ,02, ■■■ of maximal 
triangulations is also denoted by Tniax(fli ,^2, • • • ,«/)■ We denote by f (ai,a2, ■ ■ ■ 
an arbitrary weakly convex polygon with / edges of successive weights ai, a2-- ai. 
Moreover, we use an exponential notation for indicating several consecutive edges of 
weight 1: we denote for instance by P(l^,5,2) = 1, 1,5,2) a decagon with 5 edges: 
three consecutive edges of weight 1 , followed by an edge of weight 5 and a final edge 
of weight 2. We use the same notation for the number of triangulations: T6(l^,5,2) is 
the number of triangulations of f (1^,5,2) involving 6 vertices. 

3.2 Inclusion-exclusion principle 

Our first aim is the determination of the number of maximal triangulations for weakly 
convex polygons. This can be achieved by reducing the problem to the case of strictly 
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convex polygons where formulae are known. Replacing the first edge of weight cii by 
edges of weight 1 leads to the following proposition. 

Proposition 3.1 Given an integer I > 3 and I strictly positive integers a\, a2-. ai, we 
have 

Lai/2J /fli-A 
i:max(fll,fl2,---,a/) = E (^1)'' u n:max(l'''"'*',fl2,---,a/)- 

k=() \ J 

Proof. We consider the set T of maximal triangulations of ,02, . . . ,(23). A trian- 
gle of a triangulation is exterior if it involves two (necessarily adjacent) edges among 
the a\ first edges (of weight 1) of f (!"' ,a2, ■ ■ ■ There is an obvious one-to-one cor- 
respondence between the set %_ of maximal triangulations of ,02, ■■■,ai) without 
exterior triangles and the set of maximal triangulations of P{ai,a2, ... ,ai), by contin- 
uously straightening the set formed by the first ai 1— edges (see Figure |2]i. It is thus 
sufficient to enumerate . It is easier to enumerate the complementary set T \ 1^ . 



Figure 2: Straightening a maximal triangulation without exterior triangles. 

Denoting by E the set of all ai — 1 possible exterior triangles and, for A G E, Ta 
the set of triangulations containing A, the set T \ ^ is the union of the sets T^. We 
enumerate this set by the inclusion-exclusion principle; 

L«l/2J 

i-=l {Ai.A, AJ 

The upper bound [fli /2J in the summation is due to the fact that a triangulation contains 
at most [fli/2j exterior triangles. 

It remains to enumerate the intersections. Fix k < ai/2. If some triangles among 
Ai, A2... Ak have non-disjoint interiors, the intersection is empty. Otherwise, we asso- 
ciate to each element of Tai H Ta, n • • • n Ta^, a maximal triangulation of the polygon 
P{1"^^'' ,a2, . . . ,fl/) by erasing triangles Ai, A2... A^^. We also keep track of erased tri- 
angles by marking the remaining edge for each of these triangles (see Figure |3]l. This 
defines a map <Pai.A2,...,A|.- from the set T^^ nTA2 ^ ■ ■ ' l^^A^- to the set Mi^of all triangula- 
tions of P(l"i^*,a2, ■ ■ ■ ,0/) with k marked edges among the first ai edges of weight 
1. This map is obviously injective: we can reconstruct the initial triangulation by glu- 
ing triangles onto the marked edges. Moreover, the union of images of cpA[.A2....,Ati for 
all "admissible" A;— tuples of exterior triangles is clearly a disjoint union, by the same 
remark as for injectivity, and fills fMk for exactly the same reason: the reconstruction is 
unique and is always possible. Thus, 

L #(^'Ain?A,n---nTAj=#iA/^= r'^^)Tmax(l"'"^a2,...,a/). 
{A1.A2 A,} V / 

As #T = 

'tmax(l"' 7^2: • • • jCii)-, the proof is achieved. □ 
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Figure 3: Erasing some exterior triangles and keeping marks. 



3.3 Maximal triangulations 

Since combinatorial properties of triangulations of weakly convex polygons are in- 
variant under cyclic permutations of edges, we can "break up" all edges by iterating 
Proposition [3T| Hence, Xmax(fli ,^2, •••,«/) can be successively written as 

L"l/2J / _ Ir \ 

L (-i)^'("'pMw(i"'-'^',«2,. ..,«/) 



If 

ki,k2 it; \ 1=1 V 



k,.k2.-,ki \ 1=1 



k: i r'(i!=,«,-A,)-2- 



We thus obtain Tmax(ai,a27 ■ ■ ■ as a linear combination Y.i>2^jCj-2 of Catalan 
numbers. The coefficients bj are given by 

L«,/2J [a,/2\ / ' /rj -k\\ , ' 



j>2 ki=Q ki=0 \ !=1 \ / / 1=1 



with 



[m/2J / ,\ 

p,n = Li-^)'r,V'-' (2) 



/t=0 



defining the sequence {p„j)m > 1 of maximal edge-polynomials. 
We consider the generating function 



Gc(o=LQ.^=i:(2;)^ (3) 

A'>0 A:>0 ^ " ^ " ^ 



for the sequence of Catalan numbers (corresponding to the analytic expression ' ^' ). 
Given a polynomial /^(f ) = Y,k>2 ^kt'^^ we define 

{p{t)/Gc{t)), = Y^a,Ck-2- (4) 

t>2 



5 



This "umbral" notation is suggested by the fact that polynomials and formal power 
series are mutually dual. 

We have obtained the following result concerning the number of maximal triangu- 
lations: 

Theorem 3.2 Given natural numbers I > 3 and ai, a2-. ai > 1, we have 

I 



1=1 



using the notations offormulae\2\\3\and^above. 

3.4 Examples and remarks 

The first few maximal edge-polynomials are 

pi=t ps^t^ -4t'^ + 3t^ 

P2=t^-t pe = t^-5t^ + 6t^~t^ 

P3=t^-2t^ ;97 =f^-6f^+10f^-4f2 

P4^t^- 3f ^ + t^ pt,=t^- if + 15f - lOf ^ + f"^ 

Example. The two weakly convex polygons P( 1,5, 2, 3, 4) of Figure[T]have 

{P\ P5 P2 P3 P4,t^Gc{t))t 

= (f '5 - lOf 14 + 39f '3 - 75f 12 + 74f 11 - 35f 1" + 6t'^J^Gc{t))t 
= Ci3 - IOC12 + 39Cii - 75Cio + 74C9 - 35C8 + 6C7 
= 7429000 - 10 • 208012 + 39 • 58786 - 75 • 16796 

+74 -4863 -35 -1430 + 6 -429 
= 8046 



maximal triangulations. 

Remark. We have i,„ax{n,m, 1^) = (""Jj"') for all «,m > 1. 

Indeed, triangles in a maximal triangulation of P{l,n, l,m) are linearly ordered and 
in one-to-one correspondence with the n + m "segments" of the two opposite "long" 
edges. Gluing a triangle onto one of the two edges of length one, we have the formula 

w(«+i,m+i,i) = (T)- 

Remark. The function n 1 — > f(k , n) — 'Cmax(l'^^^;^) is polynomial of degree k. 
First, a classification of the triangulations of P{l^^'',n) according to the third vertex of 
the last triangle based on the edge of length n gives the formula 

k 

f{k,n)^Y.Ck-lf{l.n^l). 

1=0 

The result follows then from an induction on k. It is obvious that /(O, n) = 1 for all n. 
Suppose that f{l,n) is polynomial in n of degree I for every / < k. Using Co = 1, the 
difference 

k 

/(^+l,n)-/(A:+l,n-l)=^Q+,_,/(/,«-l) 

/=o 

is then a polynomial of degree k and a sum over n implies the result. 
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Remark. The sequence of numbers of maximal triangulations of the weakly convex 
polygons P(2,2,2), P(2,2,2,2)... starts as : 

4, 30, 250, 2236, 20979, 203748, 2031054, 20662980, 2136791 14, 2239507936... 
(see sequence A86452 of ifHll ). 

Remark. Maximal edge polynomials can also be defined recursively by po = 1, pi =t 
and pm=t{p „,_i — Pm-i) and are related to Fibonacci numbers (the closely related 
polynomials i"'^^)^'^ ^l^o called "Fibonacci polynomials"). 

3.5 Non-maximal triangulations 

An arbitrary {i.e. not necessarily maximal) triangulation of a weakly convex polygon 
P{ai,. ■■ ,ai) is a maximal triangulation of a subset involving all extremal vertices of 
the weakly convex configuration P{ai, . . . ,ai). It amounts thus to the choice, for every 
1 < ; < Z, of a number I < bi < ai and of bi — 1 points among the a, — 1 interior points 
of the th edge, followed by the choice of a triangulation of P{bi,. . . ,bi). 
The triangulation polynomial of P{ai, ... ,a[) is thus given by 

l<fo,<«,- V;=l V' V / ,=1 
= {Y{Pa,/Gc{t))^, 

where the complete edge-polynomials'p^j G are defined as 

P^n=ti^lZ^^Pk{t)sK (5) 

We have proved 

Theorem 3.3 Given natural numbers I > 3 and a\, a2... ai > 1, the triangulation 
polynomial of P{ai,a2,---,ai) is 

I 

p^{ai,a2, . . . ,ai) =Y,1k{ai,a2, . . . ,ai)s'' = {Y[Pai{t),t^Gc{t))^, 

k 1=1 

using the notations of formulae\3\^and\5\above. 

An immediate consequence is the following slightly surprising fact: 

Corollary 3.4 Enumerative properties of triangulations for weakly convex polygons 
do not depend on the particular cyclic order of edge weights. 

The first few complete edge-polynomials are 

Pi= pi s^t s , 

P2= P2 s'^ + Pl S ^ {t^ -t) S^ +t S , 

P3 = p^ s^+2p2 r + pi s^ (f^ -2f^) s^+2{t^-t) s^ + t s , 

P4 ~ PA s'^ + 3p3 + 3p2 + pi S 

= {t^-3t^ + t^) / + 3(r3-2f2) s^ + 3{t^-t)s^ + ts . 
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Example. The triangulation polynomial /7x (1,5, 2, 3, 4) of the two weakly convex poly- 
gons of Figure[T]equals 

= 8046s'^ + 372505^'* + 77467 s^^ + 95364^12 + 77048 ' 
+42776 + 16584 + 4460 / + 805 + 90s^ + 5sK 

4 Nearly convex polygons 

Nearly convex polygons are small perturbations of weakly convex polygons and form 
the correct framework for generalizing Theorem [32] and Theorem [33] We give first a 
definition of near-edges, which are small deformations of edges, and introduce nearly 
convex polygons and, using the formalism of "roofs", the associated near-edge poly- 
nomials. Then we state and prove the main theorem. A last subsection describes fac- 
torization properties of near-edges and gives a classification of small near-edges. 

4.1 Near-edges 

In order to describe deformations of an edge £ in a weakly convex polygon P = 
P{n, . . . ), we choose coordinates such that P is contained in the upper half-plane y > 
and £ is a subset of the boundary y = 0. This leads to the following definition. 
A near-edge of weight n (or an n— near-edge) is a sequence £ of n + 1 points 



(Po,A,... 




such that xo < xi < • • • < x„ and yo = yn = 0. We consider the complete order Pq<P\ < 
P2 < ■ ■ ■ < Pfi on E and call Pq, respectively P„, the initial, respectively final, vertex of 
the near-edge E. We denote a near-edge E either by the sequence E = {Pq, . . . ,f„) of 
its n + l points or by the real matrix 

^ ^ / xo XI ... x„_i x„ \ 
\ Q yi ... yn-i J 

of size 2 X (« + 1). 

A continuous deformation of near-edges, which preserves collinearity and non- 
collinearity of all triplets of points, is called an isotopy. Two near-edges joined by an 
isotopy are isotopic. 

Given a near-edge E with points P, e as above, we denote by E^ the near-edge 
with points 



n(e) = 



for < /t < «. 

In particular E = E^ and all near-edges E^ are isotopic for e > 0. 
4.2 Nearly convex polygons 

Let P be a strictly convex polygon with / extremal vertices Vb, V\, V2. . . V/ = Vo, ap- 
pearing in counterclockwise order around the boundary dP of P. 
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Given a sequence £1 , . . . ,£/ where £, = {Pi.o,PiA ,Pi.ni) is an near-edge, we 
denote by G{Ei, . . . ,Ei\P) the unique configuration obtained by gluing the «,— near- 
edge Ei, rescaled suitably by an orientation-preserving similitude, along the oriented 
edge of P which starts at y,_i and ends at V, . 

More precisely, the gluing map (p, is the unique orientation-preserving similitude 
of such that 

(?iiPifl) =Vi-i and(p,{Pi,„.) = V, . 

The configuration G{Ei , . . . ,Ei\P) is the set of points u'^j(p,(£',) C R^. 
We have the following result which we state without proof. 

Proposition 4.1 Consider a strictly convex l^gon P and I near-edges E\^. . . ,Ei, 

1. The configurations G(£'j ' , . . . ,Ef \P) are all isotopic for all e,- > small enough. 
This defines an isotopy class associated to the near-edges and the polygon. 

2. Given a second strictly convex l~gon Q, the configurations G{E^'- . . ,Ef\P) 
and G(£'j' , . . . ,E^' \ Q) are isotopic for all Ej > small enough. Hence, the iso- 
topy class above does not depend on P. 

3. The isotopy class defined in this way depends only on the isotopy classes of the 
near-edges. 

A nearly convex polygon is a configuration in the isotopy class associated by Propo- 
sition |4T| to a sequence Ei,...,Ei of near-edges. For the sake of convenience, the 
isotopy class itself is also called a nearly convex polygon. As far as combinatorial 
properties of triangulations are concerned, all the configurations of the class are equiv- 
alent and we denote by P{E\ ,Ei) any such configuration. This notation is a natural 
extension of the notation already used for weakly convex polygons, with integers n 
representing weighted edges of weakly convex polygons. 




Figure 4: Perturbating P(5,4,5) into a nearly convex polygon. 



One can think of a nearly convex polygon as a small perturbation of the config- 
uration associated to a weakly convex polygon. Figure |4] shows the weakly convex 
polygon f (5,4,5) and a nearly convex polygon obtained by moving slightly non- 
extremal vertices perpendicularly to the three corresponding edges (with a hopefully 
evident notation indicating the perturbation). This nearly convex polygon is isotopic to 
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P{Eii,Ei,,Ec) where 
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4.3 Edge-type triangles 

When considering a triangulation T of a nearly convex polygon P, and a near-edge E 
of P, some triangles of T have their three vertices in E (we identify here the near-edge 
and its realization in P). We call such a triangle A an edge-type triangle of E and we 
write A C £. The following technical lemma is the key of our main result. 

Lemma 4.2 Given a triangulation T of a nearly convex polygon P and a near-edge 
E of P, each point of E that is a vertex of 1 belongs to 3Conv(P) or to at least one 
edge-type triangle ofE. 

Proof. Recall that nearly convex polygons are "small" perturbations of weakly convex 
polygons. For the fixed particular realization of the near-edge E in the current polygon, 
denote hy E^ the realization obtained by setting all points of E closer to the edge [Pq , f „] 
by multiplying the distances to this edge by e e (0, 1). We can then replace E by E'^ 
(e G (0, 1)) whithout changing the structure of the triangulation (Proposition l4.1l i. 

Let us prove the lemma by contradiction: suppose that there exists a point M of 
E which is an interior vertex of T and which belongs to no edge-type triangle of E. 
Hence, there are at least three edges of the triangulation starting at M and forming 
adjacent angles less than n. If no such edge joins two vertices of E, a substitution of E 
by E^, for appropriately small e, moves M so close to the segment [Po,^'h] that one of 
these angles becomes greater than n. Therefore, one at least of these edges has its other 
endpoint N E E. Suppose for convenience, but without loss of generality, that M < N. 
Then consider the next segment of the triangulation starting from M, according to the 
clockwise order around M, and denote by Q its other endpoint. The asumptions on 
M imply that Q ^ E. Replacing now E by E^, for e sufficently small, gives an angle 
QMN > n for the triangle (QMN), which is a contradiction. □ 

4.4 Roofs 

Let us define the set 



which will be proved to be a piecewise linear path separating the edge-type triangles 
contained in E from the remaining triangles of T . 

The idea for counting triangulations of a nearly convex polygon is to classify trian- 
gulations according to the paths thus obtained from all near-edges and to enumerate all 
triangulations giving rise to such a set of paths. The following definition is useful for 
the description of all possibilities. 
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A partial roof with length Len{R) = ^ of £ is a piecewise linear path R starting at 
the initial vertex and ending at the final vertex of E, whose k+l vertices are elements 
of E in increasing order; Pj^ = Poj^/i 7 • • • jPjt-i i^A ^ ^^^^ partial roof is denoted 



Lemma 4.3 Given a triangulation T of a nearly convex polygon P and a near-edge E 
ofP, the set R{'T ,E) is a partial roof of E. 

Proof. Let us introduce the set P' = Conv(P) \ Uac£^ = Ua(/£^- Since Conv(P) \ 
Conv(£') is a connected subset of P' which intersects all triangles A E, P' is con- 
nected. Hence, it is a (generally non convex) polygon with vertices in T . 

Considering the inclusions Conv(P) \Conv(£') C P' C Conv(P), the boundary of 
P' coincides with the boundary of Conv(P) outside of Conv(£'). Hence, R{'T ,E) is a 
piecewise linear path with vertices PjQ.Pj^ ,Pji;_t ,Pji, in E, whose orientation can 
be chosen such that = Pq and P/^. = P„ (where Pq and P„ are the initial and final 
vertices of E). 

Let us now prove by contradiction that this sequence of points of E is increasing. 
Otherwise, consider the first decreasing step: P/.^, < Pj.. There are two cases. 

Suppose first that the point Py.^j is above the line L defined by P,. , and Pj., ac- 
cording to the standard coordinates of the near-edge E. By definition of R{1 ,E), the 
point Pj. is linked by an edge of T to a point Q £ P\E which crosses the segment 
[Py. I jPy.^J. As in the proof of Lemma 14. 21 a substitution of E by E^, for e sufficiently 
small, removes this crossing, which is in contradiction with Proposition |4T| 

Suppose now that the point P^.^j is below the line L. The point Py.^, is linked by 
an edge of ^ to a point Q' G P\E. A substitution of E by E^, for e sufficiently small, 
creates a crossing between this edge and the edge [P;,_, ,Pjj], which is in contradiction 
with Proposition |4T| □ 

The definition of a partial roof refers to the coordinate representation of E: the 
"sky" (which corresponds to the interior of the nearly convex polygon) is "above" E. 
We are interested in points and triangles "sheltered" by a partial roof. 

In order to define properly the region sheltered by a partial roof, we consider again 
coordinates (formula|6]l of the near-edge E: we define the lower boundary d^E of E 
as the piecewise-linear path d^E = (3Conv(£')) n {{x,y) \ y < 0}, which is the "low- 
est" possible partial roof. Remark that each partial roof R, and in particular the lower 
boundary, is the graph of a piecewise-affine function //; : [xo,x„] — > R. The graph 
8^£' is below each other partial roof: we have f^-^{x) < for all in [.fOi-f;,]. The 
region sheltered by the partial roof R is then the (generally non convex) subset S{E,R) 
enclosed by d^E and R : 



We use the same notations 8 E and S{E,R) for denoting the corresponding subsets of 
a realization of £ in a nearly convex polygon P. 

Lemma 4.4 Given a triangulation T of a nearly convex polygon P and a near-edge E 
ofP, T induces a triangulation on the sheltered region: 



by 



P=[P,„=P0,P;,,... 




5(£,/?) = {(x,y)|xe [.to,x, 



/a-£W<y</«(x)}. 



S{E,R{<r,E)) = U A. 
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Proof. It is a consequence of the proof of Lemma l43] □ 
A partial roof /? is a roof if £ C S{E,R). If T is a maximal triangulation of P, 
R = R{'T ,E) is a roof by Lemma l4~2l and T induces a triangulation on S{E,R) whose 
vertices are exactly the points of E by Lemma l4!4l Enumerating induced triangulations 
on S(E,R) leads to the definition of maximal near-edges polynomials. 

4.5 Near-edge polynomials 

We define the maximal polynomial of the near-edge E by 

PE^ T^maxiE,R) PLen{R) eZ[f], (7) 

R roof of E 

where Zynax{E,R) denotes the number of triangulations of S{E,R) involving exactly all 
points of E r)S{E ,R) and the polynomials pm are the maximal edge polynomials given 
by formula|2]of subsection l3.3l 

Starting from a triangulation which is not maximal, the path R induced on a near- 
edge E is in general only a partial roof and T induces not necessarily a maximal trian- 
gulation of the sheltered region S{E,R). This suggest to introduce "sub-near-edges" in 
order to deal with these difficulties. 

Let (£■) ~ Extr(£') n d^E denote the set of extremal points of the lower bound- 
ary. A k— sub-near-edge of E is an increasing subsequence £''c£'ofA:+l <n+l 
elements containing the set {E). In particular, any sub-near-edge E' of E has initial 
vertex Pq, final vertex P„ and verifies d^E' = d^E. We note E' < E. 



Figure 5: An 8— near-edge and a roof of length 4 of a 6— sub-near-edge. 
Example. The left half of Figure|5]displays the 8— near-edge 

f 123 4 5 6 78\ 
E = {Po,...,P,)=^^ -1 1 1 -2 -3 -2 -1 j • 

We have V^{E) = {Po,Pi,P5,P^) and E has 2^ sub-near-edges obtained by remov- 
ing any subset of vertices among {P2,P3,P4,P6,P7} from E. The right half of Fig- 
ure |5] shows the roof R = [Po,^'3,^'5,^'7,^'8] of the sub-near-edge E' of E defined by 
iPo,PuP2,P4,P5,Pi,P&). 

The complete polynomial 'p{E) of an «— near-edge E = (Pq, ■ ■ ■ ,Pn) is defined as 

= E E PE'^"- (8) 

m=l E'<E 

Example. We compute the complete polynomial p^^ of the near-edge 
F (1. P^ 1 2 3 4 5 \ 
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Figure 6: All roofs of sub-near-edges involved in p^^ 



involved in the nearly convex polygon P{Ea,Ei,,Ec) of Figure |4] 

Figure |6] contains all roofs of the four possible sub-near-edges of obtained by 
removing any subset of points in {Pi ,^3}. Their contributions to /J^^ are given by 



sub-near-edge 










iPQ,Pl,P2,P3,P4,P5) 




2p4S^ 


5p4S^ 


P5S^ 


{Pq,P2,P3,P4,P5) 


5p2s'' 


2/53/ 


Ipis"^ 


4 

P4S 


(P0,PuP2,P4,P5) 


5p2s'^ 


4 

pj,s 


Ipis'^ 


4 

P4S 


(P0,P2,P4,P5) 


2p\s^ 




P2S^ 


P3S^ 



They sum up to the complete near-edge polynomial of £„: 

Pe, = (14/'3 + 7P4 + P5)s^ + (10p2 + 7p3 + 2p4)s^ + {2 PI + 2p2 + P3)s^ . 

4.6 Main result 

We can now state and prove the central theorem. 

Theorem 4.5 Given I > 3 near-edges Ei, E2.- Ei, the number of maximal triangula- 
tions of the nearly convex polygon P{E\ ,Ei) is given by 

I 

i=l 

and its triangulation polynomial is defined by 

p,{P{Eu. . . ,Ei)) = ^T,(P(£i, . . . ,Ei)y = (riP£,(0,f'Gc(r)),, 

k !=1 

using the notations offormulae\3\^and\8\above. 

Corollary 4.6 The number of maximal triangulations and the triangulation polyno- 
mial of a nearly convex polygon P{Ei, . . . ,Ei) does not depend on the cyclic order of 
the near-edges Ei. 
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Proof of Theorem l4.5[ Fix a triangulation T ofP = P{Ei,E2, ■ ■ ■ ,Ei), and a near-edge 
E = Ei. We have seen in Lemma l43] and Lemma l4~4l that we can associate to £ a partial 
roof R = R{'T ,E) and a triangulation of the sheltered region S{E,R) by the edge-type 
triangles of E. We denote by E' the subset of all points of E occuring in T . The set E' 
is a sub-near-edge since 1 triangulates Conv(P) and thus involves all extremal points. 
A crucial remark is that E' C S{E,R) by Lemma l4~2l Therefore, R is a roof for E\ 
Moreover, by definition of E', the triangulation on S{E,R) = S{E' ,R) is maximal with 
respect to 

Writing = R{'T ,Ei), we obtain a triangulation of each S{Ei,Ri). We get also 
a triangulation of the complement / = Conv(P) \ U/S' (£,,/?,), whose boundary is the 
union of the roofs This triangulation is not arbitrary: each vertex is in a roof 
and a triangle in / is never of edge-type and has thus not all three vertices in the same 
roof. The triangulation induced on / yields hence a triangulation of the weakly convex 
polygon P(Len(/?i), . . . ,Len (/?/)) as can be seen on Figure|7] 




Figure 7: Triangulation of the interior region /. 

We associate thus to each triangulation of P a couple ((£';,/?,-,'r,);,To), with the 
following specifications: for each I < i < I, E[ is a sub-near-edge of is a roof 

of E'j and Ti is a triangulation of S{E[,Ri) = S{Ei,Ri), maximal relatively to E'i; Tq 
is a maximal triangulation of P(Len(/?i), . . . ,Len(/?/)). This correspondence is clearly 
one-to-one: it is easy to reconstruct a triangulation of P, given such data, and there is 
only one possible reconstruction. 

We consider first the case of maximal triangulations. They satisfy £■ = £, for all 
/ and this property characterizes maximal triangulations. Theorem 1 3 . 2 1 show s that the 
number of corresponding triangulations equals 

n'i:max(£';,/?;)^ Tmax(Len(/?i), . . . ,Len(/?/)) 
/ 

= {YlZmUEi,Ri)PLen{R,)it),t^Gc{t)), 
i=l 

for each choice of a family of roofs Summation over all possible choices of Ri and 
inversion of sum and product give the result. 

For the triangulation polynomial, according to the result above, each choice of a 
family of sub-near-edges gives the contribution 

1=1 

where the ki are the weights of the sub-near-edges Summation over all choices of 
sub-near-edges and inversion of sum and product achieve the proof. □ 
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Example. Consider the nearly convex polygon P{Ea,Ei,,Ec) isotopic to the perturba- 
tion depicted on Figure|4] Its near-edges 

2 3 4 5 \ 

-1 1 -1 j 

2 3 4 \ 

-1 1 y 

2 3 4 5 \ 

1 -1 1 y 

have complete polynomials 

Pe, = (14p3+7p4 + /'5>^ + (10/J2 + 7/73 + 2/74)/ 
+ (2pi +2p2 + p3)s^ , 

PEi, = {5p3+P4)s'^ + 2{2p2 + P3)s^ + {pi+P2)s^ 

Pe, = (10/73+7/?4 + 2/?5>5 + (3/72+13/;'3+4/74>'* 
+3(2/>2 + /73)s^ + ipi+P2)s^, 

and the triangulation polynomial pT:{P{Ea,Ei,,Ec)) of P{Ea,Ei,,Ec) is given by 

{PE,.it)pEAt)pESt),t^Gc{t))r 

= 194939ii^ + 338669ii3 + 263615i^2^ 119944*11 
+34773 + 6522 + 748/+ 42 . 

4.7 Arithmetics of near-edges 
4.7.1 Factorization 

A near-edge E = {Pq, . . . ,P„) factorizes into near-edges Ei,E2 if there exists a lower 
extremal vertex P^ G (E) such that the two near-edges defined by 

El = (Po^Pu. . . ,Pk-hPk), E2 = {Pk,Pk+\,- ■ ■ ,Pn-l,P„) 

have the property that all points of £ \ lie strictly above every line defined by two 
distinct points of Ei for ; = 1,2. We write E ^ Ei ■ E2 if the near-edge E factorizes 
with first factor Ei and second factor £2- A near-edge is prime if it has no non-trivial 
factorization. It is easy to show that every near-edge has a unique factorization into 
prime near-edges. 

Proposition 4.7 Given a factorization E = E\ ■ E2 of a near-edge E we have 

PE = PE. and Pe = Pe, ■ 

Proof. Since the nearly convex polygons P{l'',E) (with the same notation as before: k 
successive edges of weight 1) and P(l'"',£'i ,£2) are isotopic for all ^ = 2,3, . . . we have 

(t' PE,t^Gc{t)), = (t' Pe, PE,,t^Gc{t)),. 

This implies the result since det{(Ci+j^k) j^,,) > for all ^ > and n > 1 (this 
follows for instance easily from Exercice 6.26.b in lfT4l ). □ 



Ea = 
Eb = 
E,= 






1 





1 





1 





1 





1 





2 
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4.7.2 Polynomials for small near-edges 

This subsection describes all 1—, 2— and 3— near-edges up to isotopy and gives their 
polynomials. We will use the following definition: a near-edge is generic if its under- 
lying set of points is a generic configuration of R^, i.e. if three distinct points of E are 
never coUinear. 

We will also use the following obvious fact. If two «— near-edges E = {Pq, ■■■,?„) 
and are vertical mirrors, i.e. if E' = (P„,P„_i , . . . o) where P/ = ^ ^ is 

the Euclidean reflection of P, = i ^' ] with respect to the vertical line x = 0, then 



yi 



Pe = Pe 



1— near-edges The unique 1— near-edge can be represented by £1 = || ^ ). It 
is generic and prime and has complete polynomial = p\s = s t. 

2— near-edges There are two generic 2— near-edges, represented by 

^2=1= ( 10)' '^2,2=( _\ I 

£2,1 is prime while £2,2 ^E\ - Ex. They have complete polynomials 

P£2,i =P2S^+P\S, PE22 = {P2+P\)s^ = Vei ■ 

Moreover, there is also a unique non-generic 2— near-edge represented for instance 

by 

/ 1 2 

£2,3 = 







with complete polynomial given by 



PE2.i ^ P2'^^ + Pl^ = PE2,i ■ 

3— near-edges There are eight generic 3— near-edges represented by 





The first five are prime. The last three have factorizations 
£3,6 = E1E2.1 ,£3.7 = Ej, £3,8 = E2.1E1 
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The pairs {£3,1, £3, 3}, {£3 4, £3 5}, {£3 5, £3 8} are vertical mirrors. The prime near- 
edges have complete polynomials 

^£3,1 = Fej.j = i.P2 + Pi)s^ + 2p2«^ + pi* , 
Pi-3,4 = ?i?3,5 = (2P2 +/?3)«Hp^^ • 

There are moreover nine more 3— near-edges which are not generic. They are rep- 
resented for instance by 



Ei. 
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£3,9 - ^ 2 j ^3.'° = 

£3.11-1 1 ] ^^ '2- 
/ 1 2 3 \ 








1 




2 


3 







1 




-2 








1 




2 


3 












-1 












1 


2 


3 


1 







1 















1 




2 


3 







2 




1 






The following near-edges factorize: 

£3,10 =£'2,3 £l: £3,16 =£1 £2,3 

The remaining near-edges are prime and have complete polynomials 



^£3,9 


= PEiAS 




+ p\s, 




= PE3A3 


= {P2+P3)sU 


■2p2S^ + pis 


^£3.12 


= PE3,4 


= (P2+/'3)^^4 


■P2S^, 


^£3.17 


= P3 


= P3S^ +2p2S^ 


+ pis . 



5 Remarks and questions 

5.1 Choice of the triangulation polynomial 

One can also consider the triangulation polynomial defined by 

ka.ki 

counting the number of triangulations using ko vertices and ki edges. The number k2 
of triangles can then be recovered using the Euler characteristic ^0 — ^1 + ^2 = 1 of 
a compact, simply connected triangulated polygonal region in r2. This more general 
polynomial yields the same information as the complete polynomial considered above 
except if the boundary 3(Conv(c)) contains points of C which are not extremal. Most 
of the results and algorithms can easily be modified in order to deal with this more 
general polynomial. For clarity and concision we described here the simpler version 
defined above. 
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5.2 General configurations and nearly convex polygons 

Remark that every generic configuration is isotopic to a nearly convex polygon. Indeed, 
every extremal point Q of a generic configuration C yields a realization of C as a nearly 
convex polygon with two trivial near-edges (each consisting of Q and of a neighbouring 
extremal point) and a near-edge defined by C \ {Q}, which is unique up to isotopy. 

However, the framework of near-edges is not interesting for a general generic con- 
figuration. It speeds up computations only in the case where the configuration has a 
"non-trivial" factorization into near-edges. 

5.3 Remarks on effectiveness 

Near-edge polynomials are in general difficult to compute. We will present a few algo- 
rithms dealing with them in a further paper One of these algorithms is a slightly more 
sophisticated version of an algorithm by Kaibel and Ziegler described in |9| and yields 
also a general purpose algorithm (unfortunately of exponential complexity), for com- 
puting arbitrary triangulation polynomials. This algorithm, based on a transfer matrix, 
is fairly simple and it would be interesting to compare its performance with existing 
algorithms, like for instance the algorithm of Aichholzer described in IT]. 

The next subsection describes a family of near-edges for which the computation of 
near-edge polynomials is much easier and can be achieved by an algorithm of poly- 
nomial time-complexity. A detailled description of the algorithm will be given in our 
planned future paper 

5.4 Convex near-edges 

A near-edge E ~ {Pq, ■ ■ ■ iPn] is convex if fo, . . . ,f„ are extremal points of Conv(£'). 
Otherwise stated, the points of a convex near-edge are the vertices of a convex polygon 
with (n + 1) edges. There are thus exactly 2"^' non-isotopic convex «— near-edges. 
A convex near-edge can be represented by a sequence of points 

"-(o )•■■■•"- (..>-.,) "-(S) 

where ei,...,e„_i G {±1}. There are 2"^' equivalence classes of convex «— near- 
edges, encoded by «— tuples (ei , . . . ,e„_i ) in {±1}"^', The convex near-edge with 
e,- = — 1, for all /, has the factorization E". All others are prime. 

A future paper will describe an algorithm having polynomial time and memory 
requirements for computing maximal and complete edge-polynomials of convex near- 
edges. It provides an efficient method for counting triangulations of nearly convex 
polygons involving only convex near-edges. Completing each convex n — 1 —near-edge 
with two trivial near-edges, we get an exponentially large class of configurations for 
which the problem of counting triangulations can be solved in polynomial time. 

5.5 Convex near-edges related to the Legendre symbol 

We used the Legendre symbol to produce data for testing our algorithm. The surprising 
results lead to formulate the conjecture below. 

Given an odd prime p, the Legendre symbol, denoted by e {il} for 1 < < 
p—l defines a non-trivial homomorphism between the multiplicative groups (Z/pZ)* 
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and {±1}. It can be computed using quadratic reciprocity or the equality 

Ejc^P-i'/^ (mod;?). 



We consider two convex {p + 1 )— near-edges Ep,Ep associated to the sequences 
III I I ^jjj _ I I I I I 



P/ \PJ \ P J \PJ \PJ \ P 



of (negated) Legendre symbols. For p = 3 (mod 4) the identity j = — y-jj-j im- 
plies that Ep and E^ have identical (complete) triangulation polynomials. 

Computation of the maximal triangulation polynomials p^+ and p^- for all odd 
primes p < 200 suggests: 

Conjecture 5.1 Using the notations of fonnulae\3\and^ we have 

for P a polynomial of the form t p^^ , t p\._ or t Pp-+ Pf-- 

p p ^ ^ 
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